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BACKGROUND 


selection  process,  as  described  above,  ranges  from  nine  to 
23  months.  A significant  portion  of  this  time  is  involved  in 
working  with  the  bt'nchmark.  The  system  we  are  describing 
here  is  designed  to  nxluce  this  time. 


The  acquisition  of  major  Automatic  Data  Pnici-ssing  Equip- 
ment (ADPE)  systems  in  the  Department  of  the  Navy  is 
accomplished  by  either  single  source  acquisition  or  com- 
petitive selection.'  In  the  case  of  the  single  source  acquisition, 
only  one  vendor  is  considered  as  having  the  system  capable 
of  satisfying  the  needs  of  the  procuring  organisation.  The 
criteria  which  must  be  satisfied  for  a single  source  acquisition 
include  the  need  for  unique  hardware,  excessive  conversion/ 
reprogramming  cost  and/or  time,  etc.  The  competitive  se- 
lection, on  the  other  hand,  is  open  to  any  vendor  who  feels 
he  can  provide  a system  meeting  the  requirements  specified 
in  the  solicitation  document  associated  with  that  procun*- 
ment. 

The  major  steps  in  the  competitive  process  are,  briefly, 
as  follows; 

(1)  The  requestor  prepares  a solicitation  document  which 
explains  the  requirements  of  the  system  being  pro- 
cured. 

(2)  Benchmark  programs  are  prepared.  These  will  be  used 
to  ascertain  that  vendors  participating  in  the  compe- 
tition can  meet  minimal  performance  standards.  These 
programs  may  be  written  in  various  higher  level 
languages.  We  will  restrict  our  discussions  to  COBOL 
programs. 

(3)  The  vendor  examines  the  solicitation  document  to 
determine  whether  he  has  a system  capable  of  meeting 
the  requirements. 

f ^ W The  vendor  obtains  the  benchmark,  converts  it  to  the 
system  being  considered,  and  determines  if  the  hard- 
ware/software  will  be  price-competitive  with  the  sys- 
terns  most  likely  to  be  offered  by  his  competitors. 

(8)  The  vendor  must  then  demonstrate  that  the  execution 
of  the  benchmark  can  be  accomplished  within  the 

Stime  specified  in  the  solicitation  document. 

(6)  The  award  is  generally  made  to  the  vendor  who 
qualifies  with  respect  to  timed  benchmarks  and  offers 
to  provide  the  system  which  meets  the  needs  of  the 
user  at  the  lowest  overall  cost  to  the  Government. 

The  usual  amount  of  time  involved  in  the  competitive 


USE  OF  BENCHMARKS  IN  THE  SELECTION 
PROCESS 


The  benchmark  is  a vital  part  of  the  competitive  selection 
process.  It  becomes  the  t<H)l  for  minimum  measurement  to 
be  used  against  all  systems  being  considered.  Therefore  it  is 
important  that  the  benchmark  b<>  constructed  in  such  a way 
as  to  accurately  mflect  the  system  requirements  being  speci- 
fied. The  system  requirements  an-  defined  in  terms  of  the 
cum-nt  workload  and  the  projected  future  workload.  Properly 
prepared  benchmarks  will  demonstrate  that  the  system  being 
offeri-d  contains  adequate  memory  and  p<-ripherals,  and  that 
the  throughput  sp<H-ds  are  sufficient  to  process  the  projected 
future  workload.  Additionally,  this  exercise  demonstrates 
that  the  operating  system  and  supporting  software  are 
operative.  From  a functional  standpoint,  the  "ideal  bench- 
mark” situation  could  be  described  as  follows: 


(1)  The  programs  would  be  coded  using  the  language 
elements  defined  in  the  American  National  Standard 
('OBOL,  BO  that  source  code  convemion  is  minimal. 

(2)  The  implementation  of  the  benchmark  programs  by 
the  various  vendors  could  be  monitored  as  in  a con- 
trolled environment.  This  would  provide  useful  infor- 
mation as  to  the  impact  of  converting  other  programs 
after  the  new  system  is  delivered. 

(3)  The  programs  have  been  debugged  to  the  extent  that 
they  will  give  predictable  results  on  both  the  native 
and  the  target  machines. 

(4)  The  data  files  would  be  in  a form  readily  acceptable 
to  all  systems,  but  would  at  the  same  time  be  con- 
sistent with  any  given  system’s  architecture,  so  that 
there  is  no  loss  of  efficiency,  or  validity  of  results. 

(5)  The  checking  of  benchmark  processing  results  would 
be  as  automated  as  possible. 

Unforturwtely,  this  ideal  situation  seldom  exists.  This 
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results  in  excessive  time  and  cost  expended  on  the  part  of 
the  vendor  in  procc'ssing  the  benchmark.  It  is  not  unusual 
for  a vendor  to  spend  six  to  nine  calendar  months  just  pre- 
paring the  Is'iichaiark'  picgrams  for  processing,  or  for  the 
cost  of  pn>cessing  them  to  represent  10  percent  or  more  of 
the  eventual  bid  price.  The  cost  and  time  related  to  processing 
a benchmark  causes  vendors  to  be  more  s*-lective  in  res|K>nd- 
ing  to  requests  for  proposals  (RPPs).  This  could  result  in  the 
Is-st  system  not  being  offered  if  the  vendor  feels  he  may  not 
have  a good  chance  of  winning. 


I'HKVIOUS  BENt'H.MARK  EFFORTS 


Little  has  been  done  in  the  area  of  making  the  benchmark, 
and,  as  a result,  the  competitive  process,  more  palatable.  In 
the  past,  various  methods  have  been  used  in  presenting  the 
hs'iichmark  to  the  vendor.  These  range  from  the  attitude  of 
“here  is  the  bt-nchmark,  do  with  it  what  you  must  in  order 
to  make  it  run  on  your  system,  and  in  the  meantime  don’t 
ls)ther  me”  to  a recent  instance  where  the  benchmark  and 
its  related  data  were  provided  in  what  could  be  described  as 
“machine  interchangeable”  form  (i.e.,  all  data  was  in  DIS- 
PLAY form,  several  "dangerous”  language  features  were 
not  used,  etc.).  This  approach  is  consistent  with  recommen- 
dations made  by  Meltser  and  Ickes.* 

The  effect  of  the  “don’t  care”  philosophy  is  that  the 
vendor  is  |>ermittcd  to  make  any  changes  he  desires  in 
implementing  the  benchmark.  This  may  destroy  the  repre- 
sentativeness of  the  benchnuu’k.  At  the  same  time,  through 
the  use  of  his  most  talented  programmers  and/or  analysts, 
a vendor  could  optimise  the  programs  for  faster  execution. 
It  is  often  the  case  that  the  talent  the  vendor  is  able  to  turn 
l(sjse  on  the  benchmark  is  superior  to  that  of  the  average 
user  r>rganixation.  Therefore,  the  credibility  of  the  benchmark 
is  somewhat  lessened,  and  the  timing  results  now  represent 
more  what  the  vendor’s  programming  staff  can  do  than 
what  was  originally  intended. 

The  philosophy  behind  the  machine  interchangeable 
COBOL  calls  for  no  modification  being  permitted  to  the 
source  progranu  except  in  the  Environment  Division.  Basi- 
cally, the  idea  of  machine  interchangeable  COBOL  is  to 
eliminate  any  form  of  data  representation  except  for  standard 
data  format  (e.g.,  character  representation  only — no  binary, 
packed  decimal,  floating  point,  etc.). 

The  machine  interchangeable  approach  satisfies  the  ma- 
jority of  the  criteria  described  in  presenting  the  ideal  bench- 
mark; however  there  are  two  comments  worth  making; 

(1)  'The  decision  to  stay  with  the  standard  data  format 
may  force  several  vendors  not  to  participate;  pri- 
marily those  writh  systems  not  capable  of  character 
addressing  and/or  decimal  arithmetic. 

(2)  The  resources  used  in  manually  converting  the  bench- 
mark package  to  machine  interchangeable  format  will 
include  a substantial  amount  of  manpower  and  com- 
puter time. 


THE  SANITATION  EFFORT  BY  THE  U.S.  NAVY 

The  Software  D('velopm«‘nt  Division  of  the  Depart  nient 
of  the  Navy  Automatic  Data  I’roeeaMing  Equipment  Seli'ction 
Office  (ADI’ESO)  is  hsiking  into  methods  and  techni((ues  for 
decreasing  the  amount  of  time  required  for  competitive 
selection,  and  lowering  the  overall  cost  of  the  procurement. 
Armed  with  the  knowhdge  of  problems  a.s.sociati'd  with 
benchmark  techniques,  and  in  particular  the  problem.s  a.s- 
sociated  with  natural  benchmarks,  an  effort  was  established 
to  mechanize  the  preparation  of  natural  benchmarks. 

The  vehich;  we  chose  (for  ease  of  implementation,  with 
the  necessary  documented  controls)  was  the  VB-Houtine 
developed  by  the  Navy  for  automatically  resolving  imi>Ie- 
mentation  names  within  COBOL  programs,  and  generating 
the  necessary  operating  syst<‘m  control  curds  to  compile  and 
execute  th(;  programs.* 

Each  source  program  must  be  purged  of  nonstandard 
language  elements.  This  is  accomplished  by  automatic  con- 
version (where  possible  through  simple  syntax  replacement) 
and  hand  tailoring  when  additional  logic  may  be  requin-d 
to  handle  semantic  differences  between  two  statements.  .A.lso, 
all  implementor  names  must  be  resolved,  or  convertixl  to  an 
intermediate  form.  This  results  in  COBOL  source  pmgrams 
that  are  VP-Routine  sensitive  in  that  they  can  be  tailored 
to  a given  systems  ri'quirements  by  a single  pass  of  the 
VP-Routine. 

All  data  files  necessary  for  input  to  bimchmark  programs, 
or  output  files  provided  for  verification  purposes,  must  be 
carefully  extracted  from  the  native  system  and  transformed 
into  a form  readily  acceptable  to  other  systems.  This  must 
be  done  with  no  loss  of  data  integrity. 

After  the  source  programs  and  data  have  been  converted, 
it  is  important  to  insure  that  the  execution  of  the  converted 
benchmark  programs  give  the  same  results  as  the  original 
pixigrams,  and  that  the  data  has  not  been  adversely  affi^ted. 
This  is  accomplished  by  executing  the  sanitised  benchmark 
using  the  sanitised  data  on  both  the  current  computer  system 
and  on  other  target  systems.  During  the  execution  of  the 
benchmark  it  is  useful  to  be  able  to  determine  the  degri'c  to 
which  the  data  is  exercising  the  various  procedures  of  each 
program.  Through  the  help  of  a software  monitor,  infor- 
mation is  provided  which  will  help  to  further  determine  the 
adequacy  of  the  benchnuvk. 

The  result  of  the  benchmark  preparation  process  would  be 
a viable  product  that  is: 

(1)  Well  documented. 

(2)  Cheeked  out  on  more  than  one  system. 

(3)  Easily  implementable  through  the  use  of  the  VP- 
Routine. 

A discussion  of  the  overall  system  follows  in  five  major 
sections: 

Source  Program  Preparation 
Data  Portability 
Program  Monitor 
Packaging  and  Distribution 
Limitations 
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SOURCE  PROGRAM  PREPARATION 

The  source  proftrams  that  are  to  make  up  the  benchmark 
are  pr(K’ca«“d  by  a COBOL  to  COBOL  translator  that  per- 
forms three  major  functions. 

(1)  The  Environment  Division  is  rendered  VP-Routine 
sensitive  by  the  replacement  of  all  implementf>r  names 
with  an  encoded  mnemonic  that  has  meaning  to  the 
VP-Routine,  These  mnemonics  are  later  used  in  pre- 
paring the  programs  for  a given  system. 

(2)  The  source  code  is  examined  for  nonstandard  coding, 
and  tran.slated  when  appropriate.  Where  tran.slation 
cannot  be  accomplished,  the  translator  flags  the 
offending  code. 

(3)  Based  on  parameter  cards,  the  file  descriptions  of  the 
files  to  be  converted  from  the  native  system  are  used 
to  create  an  intermediate  work  file  containing  pseudo 
file/record/field  dt'seriptions.  This  file  is  later  used  to 
create  Data  Translation  programs.  This  is  fully  dis- 
cussed in  the  section  entith-d  Data  Portability. 

The  translation  of  one  COBOL  dialect  to  another  is  con- 
ceptually simple.  The  mort^  serious  problem  involves  the 
moving  of  data  from  system  to  system.  A detailed  discu.ssion 
of  this  problem  and  a suggested  solution  follows. 

DATA  I*ORTABILITY 
Problem 

The  differences  in  the  internal  representation  of  data 
among  computer  systems  represent  the  major  limitation  of 
software  portability.  These  differences  can  be  broadly  seg- 
mented into  two  categories:  Differences  in  the  character 
code  used  (i.e.,  EBCDIC,  BCD,  HELDATA,  etc.),  and 
differences  in  the  representation  of  numeric  data.  Data 
translators  for  character  code  conversion  are  widely  available, 
or  can  be  easily  created.  Transferability  of  the  second  type 
requires  more  than  simple  code  conversions,  and,  therefore, 
presents  the  greater  problem.  Furthermore,  the  specific 
representation  used  within  this  general  category  will,  for  a 
given  computer,  seriously  impact  the  effectiveness  with  which 
that  computer  is  used.  Thus,  the  system  described  here 
concerns  itself  only  with  the  conversion  of  noncharacter 
data  from  any  ‘‘native’’  computer  to  any  "target”  computer, 
in  such  a way  that  the  target  computer  architecture  is 
properly  utilised. 

Generally,  COBOL  data  portability  is  impacted  by  vari- 
ations in  numeric  data  representation,  alignment  of  data 
within  a defined  data  unit  (e.g.,  a word),  and  the  position 
and  representation  of  arithmetic  signs.  There  are  several 
forms  of  numeric  data  representation,  of  which  binary  and 
packed  decimal  are  the  most  common.  'The  packed  decimal 
format  is  not  universal  and  may  therefore  have  to  be  con- 
verted to  a completely  different  type  of  data  structure.  Data 
in  binary  representation  are  universal,  and  although  sign  con- 
ventions and  word  sise  do  vary,  conversion  between  binary 


reprcsc-ntation.s  is  relatively  simple.  Alignment  variation.s 
affect  the  positioning  of  data  within  storage  units,  particu- 
larly in  word-oriented  computers.  For  pro|)er  transformation 
of  data  from  external  storage  to  internal  storage,  thc>  ('{tBOL 
program  definition  of  this  data  must  lx-  consistent  with  the 
expected  data  position  and  alignment  on  the  external  files. 

Solution 

Oenerality  and  impartiality  are  prineipal  design  goals  of 
our  effort.  Our  system  must  p<>rform  data  translation  from 
any  given  system  to  any  other  system.  Furthermore,  we 
must  not  pt'nalize  the  architecture  of  the  target  s\str‘m. 
Generality  implies  that  the  translation  programs  must  be 
automatically  generated,  as  opposed  to  hand-coded.  Im- 
partiality means  we  must  go  fn>m  a machine  dependent 
form  to  another  machine  depr-ndent  form.  G»hk1  stmse  sug- 
gests we  do  this  thn>ugh  an  intermediate  machine  inde- 
pendent c<idc. 

We  use  available  software  as  much  as  po.ssible.  This  is 
done  by  using  the  code  conversion  subroutines  aln^ady  present 
in  a system’s  compiler,  together  with  the  data  descriptions 
in  the  COBOL  programs  being  converted.  The  data  trans- 
lators which  constituti"  the  heart  of  the  system  are  auto- 
matically generated  COBOL  program  segments.  These  data 
translators  are  used  to  convert  native  machine  dependent 
data  (.MDD)  to  standard  data  format  (SDF),  and  the  latter 
to  target  machine  dep<>ndent  data,  which  we  refer  to  as 
machine  ANSI  data  (MAD).  If  charatder  code  translation  is 
required,  it  can  lx-  (x-rformed  on  the  SDF,  since  this  data 
is  simply  a string  of  characters. 

Program  creation 

Data  translation/verification  programs  (henceforth  re- 
ferred to  simply  as  data  iranelators)  are  creab'd  from  the 
file/record  descriptions  of  the  COBOL  programs  being  con- 
verted. The  data  tran.slators  will  contain  the  following 
COBOL  file  descriptions  (FD’s): 

(1)  Machine  dependent  data  (MDD)  file  descriptions, 
which  are  those  used  to  process  the  file  on  the  native 
machine. 

(2)  Standard  data  format  (SDF)  file  di-scriptions,  in 
which  all  data  items  are  in  DISPLAY  mode,  unsigned 
and  unsynchronised. 

(3)  Machine  ANSI  data  (MAD)  file  descriptions,  in  which 
all  data  items  are  described  in  Standard  ('OBOL 
formats.* 

(4)  Machine  ANSI  data  for  the  target  machine  (MADT) 
file  descriptions,  'rhis  file  description  is  identical  to 
(3)  above,  and  is  used  for  file  comparison  purposes. 
'This  comparison  process  is  more  fully  described  below. 

Source  code  MDD  item  descriptions  which  are  not  Stand- 
ard COBOL  will  be  defined  by  the  MAD  file  description  in 
a form  which  is  as  close  to  the  native  file  description  as 
possible  (e  g.,  COMPUTATIONALr3  wUl  become  COMPU- 
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MDD 

01  MDD-RECX)RD. 

02  ALPH.\NUMERIC-D 
02  UNSIGNED-D 
02  8IGNED-D 

PICTT  RE  Xi20) 
PICTI  RE  fl(6) 
PICTURE  89(6). 

JUSTIFIED  RIGHT. 
CO.MPl  TATIOXAL-3 

SDF 

01  SDF-RECORD. 

02  ALPHANUMERIC-X 
02  UX8IGNED-X 
02  SIGXED-S 
02  SIGXED-X 

PICTI  RE  X(20). 
PICTURE  9(6). 
PICTURE  X. 
PICTURE  9(6). 

MAD 

01  MAD-RECORD. 

02  ALPHANUMERIC-A 
02  UNSIGNED- A 
01  SIGNED-A 

PICTURE  X(20) 
PICTTTRE  9(6) 
PICTURE  89(6). 

JUSTIFIED  RIGHT. 
COMPUTATIONAL 

Figure  1 — Ejuunple  of  remrd  deecription  ueed  in  a data  tranalator  program 


TATIONAL).  Figure  1 illustrates  a DATA  DIVISION  from  Data  validation  and  verification  procedures  are  also  gener- 

a data  translation  program.  Procedures  for  translating  from  ated.  Figures  2 and  3 give  examples  of  the  various  COBOL 

one  data  form  to  another  and  for  file  comparison.*  (for  data  procedures  used  for  data  translation,  validation,  and  verifi- 

verification  purposes)  are  generated  for  each  elementary  field  cation  (the  latter  two  functions  are  described  below), 

of  the  record.  There  are  three  data  translation  procedure  Once  all  the  procedures  and  file  descriptions  have  be«'n 

types,  corresponding  to  alphanumeric  data,  signed  numeric  generated,  they  are  merged  with  appropriate  housekeeping 

data,  and  unsigned  numeric  data.  The  type  of  procedure  COBOL  statements,  resulting  in  data  translators  which  are 

generated  is  based  on  the  elementarj-  COBOL  item  de-  complete  COBOL  programs  in  V'P-Routine  sensitive  format, 

scription. 

Data  translation  procedures  for  alphanumeric  and  unsigned 
numeric  data  require  no  more  than  a COBOL  MOVE  Program  optratwn 
statement.  Any  changes  in  the  data  code,  data  alignment, 

or  storage  allocation  required  in  converting  from  one  form  Since  the  native  source  code  file  descriptions  in  the  data 

to  another  are  performed  automatically  by  the  code  generated  translators  may  not  be  acceptable  on  the  target  compiler, 

(for  the  MOVE  statement)  by  the  compiler  being  used.  the  VP- Routine  is  used  to  provide  the  capability  of  selecting 

To  take  into  account  the  various  sign  conventions,  COBOL  the  appropriate  source  coding  for  use  on  the  desired  system 

procedures  are  added  to  check  the  characteristics  of  signed  (target  or  native).  This  is  accomplished  by  parameter  cards 

numeric  data.  If  translation  is  from  a machine  dependent  to  the  VP-Routine.  If  any  minor  updating  to  the  source 

form  to  a 8DF  form,  the  appropriate  sign  is  stored  as'  a programs  is  required,  this  capability  is  also  available  through 

separate  character  in  the  SDF  data  description.  If  we  are  the  VP-Routine. 

performing  the  reverse  process,  we  first  generate  the  positive  Parameters  are  used  as  input  to  the  data  translators  in 

value  of  the  machine  dependent  data  item  (through  a MOVE  order  to  direct  the  flow  of  execution.  The  three  categories  of 

statement),  then  check  the  separate  sign  character  in  the  functions  which  may  be  performed  are  data  translation, 

SDF  description,  and  if  minus  multiply  the  machine  de-  data  verification,  and  data  validation, 

pendent  data  item  by  minus  one  to  give  it  the  correct  sign.  Four  modes  of  translation  are  available.  The  mode  required 


MOVE  ALPHNITMERIC-U  TO  ALPHNTMERIC-X. 
MOVE  UNStONEO-U  TO  UN8IGNED-X. 

IF  8IONED-D  NEGATIVE 
MOVE  TO  8IGNED4 
ELSE 

MOVE  ■'+”  TO  81GNED^. 

MOVE  SIGNED-D  TO  8IGNED-X. 

MOVE  ALPHNl’MERIC-X  TO  ALPHNVMERIC-A 
MOVE  UN8IGNED-X  TO  VNSIGNED-A. 

(SDF  to  MAD  MOVE  8IGNED-X  to  810NBD-A. 
proesdurs)  IP  81GNED-8  EQl’AL  TO 

MULTIPLY  -I  BY  8IONED-A 

Figara  2— Eiampio  of  data  tranalatioo  proesduraa  uaad  in  a Data  Tranalator 
Pragram 


(MDD  to  8DF 
pneadura) 


a 

1 
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(Data  Validation 
procedure) 


IF  UN8IQNED-X  NUMERIC  NEXT  SENTENCE  ELSE 
MOVE  UNSIGNEU-X  TO  PRT-FIELD-DATA 
MOVE  "UNSIONED-X"  TO  PRT-FIELD-NAME 
PERFORM  PRINT-VALIDATION-ERROR. 


IF  UNSICNED-A  NOT  EQUAL  TO  UNSIGNED-D 
MOVE  UNSIGNED-A  TO  FLDA-NITMERIC-18V 
MOVE  UNSIGNED-A  TO  FLDA-NUMERIC-V18 
MOVE  “UNSIGNED-A”  TO  FIELD-NAME 
(DaU  Verification  MOVE  UNSIGNED-D  TO  FLDB-NUMERIC-18V 

procedure)  MOVE  UNSIONED-D  TO  FLDB-NUMER1C-V18 

PERFORM  PRINT-VERIFICATION-ERROR. 


Figure  3— Example  of  data  validation  and  data  verification  procedurea  used  in  a data 
translator  program 


I 
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ia  indicated  by  the  following  parameter  cards: 

CONVERT  MDD-8DF 
CONVERT  8DF-MAD 
CONVERT  MDD-MAD 
CONVERT  MAD-8DF 

The  first  mode  is  applicable  to  the  native  compiler,  and 
tranaiates  Machine  Dependent  Data  to  Standard  Data 
Format.  The  second  mode  ia  applicable  to  the  target  com- 
piler, and  translates  Standard  Data  Format  to  Machine 
ANSI  Data.  The  last  two  data  translation  modes  are  used 
to  create  files  for  data  verification. 

There  are  three  modes  of  data  verification.  The  specific 
one  required  is  indicated  by  one  of  the  following  parameter 
cards: 

COMPARE  MDD-MAD 
COMPARE  MAD-MADT 
COMPARE  8DF-MAD 

The  first  mode  of  data  verification  ia  used  to  compare 
Machine  ANSI  Data  files  to  Machine  Dependent  Data  files. 
The  second  mode  is  used  to  compare  two  Machine  ANSI 
Data  files.  The  last  mode  ia  mainly  for  flexibility,  and  per- 
forms a SDF  to  MAD  translation  before  a MAD  to  MADT 
comparison  is  made. 

The  third  functian  performed  by  the  eonvetsion  system 
is  data  validation.  This  consists  of  verifying  that  the  data 
content  ia  consistent  with  its  class  characteristies  (Le.,  nu- 
merically described  fields  should  contain  only  numeric  data). 
This  function  is  performed  automatically  in  combination 
with  the  translation  function,  or  during  a separate  pass, 
using  a VALIDATE  SDF-DATA  parameter. 


Data  taUdatioH/miJIeatum 


In  order  to  perform  a comparative  evaluation  of  the  per- 
formance of  computer  systems  through  the  use  of  natural 
benchmarks,  sre  must  ensure  that  the  same  anxiunt  of 
pmrssdni  completed  by  competing  systems,  and 
that  the  accuracy  of  computations  is  witto  allowable  bminds. 
Data  compariaon  and  validation  pmeedums  are  included  in 


our  system  for  this  purpose.  Additionally,  these  procedures 
provide  the  benchmark  recipient  with  a tool  to  check  the 
various  stages  of  a multi-step  processing  application  for  any 
processing  inconsistencies.  Finally,  the  procedures  are  used 
to  confirm  that  data  integrity  ia  not  lost  in  either  the  pro- 
gram conversion  or  data  conversion  process. 

Processing  integrity  is  verified  in  two  ways  through  the 
authentication  of  data  files  and  comparisons  of  files  after 
program  execution.  The  authentication  of  data  files  consists 
of  validating  the  data  item  content  for  conformance  to 
their  described  data  class  (i.e.,  numeric  fields  contain  the 
data  values  0 through  0 and,  possibly,  a sign).  This  specific 
feature  was  incorporated  in  the  system  because  it  has  been 
found,  for  example,  that  some  compiler  implementations 
permit  spaces  as  data  in  numeric  fields,  or  maintain  signed 
data  in  fields  described  as  unsigned,  and  provide  the  ap- 
propriate translation  before  processing;  other  implementa- 
tions do  not.  Validation  would  point  out  these  potential 
problem  areas. 

The  compariaon  of  files  after  program  execution  provides 
a means  of  determining  not  only  that  all  the  processing  was 
done,  but  also  that  the  numerical  results  of  this  processing 
are  within  the  accuracy  limits  allowed.  When  any  data 
discrepancy  is  found  by  the  data  translators,  a report  of  the 
discrepancy  is  produced.  A report  is  made  for  each  field  in 
error,  and  includes  the  name  of  the  field  as  defined  in  the 
program,  its  data  content  (in  the  case  of  a compariaon,  the 
field  being  compared  to  and  the  comparing  field  are  both 
displayed),  the  position  of  the  field  in  the  record,  and  relative 
record  position  in  the  file.  Record  and  error  counts  are  also 
provided  in  the  report.  Figure  4 gives  an  example  of  the 
report  generated. 


PROGRAM  MONITOR 


One  of  the  principal  concerns  in  using  benchmarks  as  a 
means  of  evaluating  computer  performance  is  whether  they 
provide  an  adequate  representation  of  the  user’s  workload, 
and  whether  they  properly  reflect  his  future  processing  needs. 
This  problem  is  not  completely  resolvable,  ^t  an  irtdication 
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DATA  VALIDATION7VERIFICATION  REPORT 


UXJICAL 

REaiRD 

FIELD 

NAME 

STARTING 

POSITION 

FIELD 

SIZE 

DATA  CONTENTS 

OOOOOH 

INSIGNED-A 

0021 

0006 

+000000000000000443.000000000000(X^^  (INCORRECT) 

• 

. -f<l0000000000u000444.000000000000000000  (CORRECT) 

0000-20 

ALPHNIMERIC-A 

0001 

0020 

SSSSSSSSodsSMgjMM 

• ••  •• 


MACHINE  ANSI  RECORDS-004M2 
MACHINE  DEPENDENT  RECORDS- 004 jW2 
ERROR  COCNT-0002 


Fiaure  4 -Sample  validation/verification  report  from  a data  tranalatnr  pmaram 


of  the  procetwiiiK  characteristics  of  the  programs  provided 
for  the  benehnutrk  can  be  of  value  in  the  evaluation  process. 
This  data  is  obtained  through  a program  execution  monitor. 

Following  program  and  data  conversion,  and  before  distri- 
bution of  the  benchmark  to  the  vendors,  this  monitor  is 
applied  to  the  benchmark  programs.  The  monitor,  written 
in  COBOI,,  inserts  control  statements  into  the  benchmark 
programs.  Execution  of  the  benehnutrk  programs  with  a 
given  set  of  data  provides  a histogram  of  procedure  activity 
in  the  pn>grams.  This,  in  turn,  can  be  used  to  determine  the 
suitability  of  the  benchmarks  in  representing  the  user  work- 
load. 

PACKAGING  AND  DISTRIBUTION 

Once  the  benchmark  has  been  sanitised  and  run  on  the 
native  system  to  be  assured  that  processing  integrity  has 
been  maintained,  the  benchmark  package  is  prepared  for 
distribution.  The  package  includes  a source  program  library, 
benchmark  data,  and  documentation. 

The  source  library  (population  file)  will  contain  the  bench- 
mark  programs,  data  translator  programs,  the  VP-Routine, 
and  the  operating  system  control  language  for  the  major 
cmnputer  systems.  The  VP-Routine  selects  the  programs 
from  the  population  file,  transforms  VP-Routine  sensitive 
progranu  to  nuM;hine  dependent  programs  by  satisfying  all 
implementor  defined  names  in  the  source  program,  and 
prepares  the  job  control  stream  for  submission  to  the  oper- 
ating system. 

Data  files  for  the  benchmarks  are  distributed  to  the  vendor 
on  magnetic  tape,  in  8DF  format. 

Documentation  pertaining  to  the  programs,  data,  instruc- 
tions for  implementation  on  a users  system,  and  all  infor- 
mation necessary  to  run  the  benchmark  for  a live  test 
demonstration  is  included  in  the  package.  This  includes  the 
following  information : 

(1)  OosB  reference  to  data  files  by  reel  number. 

(3)  Cross  reference  to  data  files  by  program. 

(8)  Cross  reference  to  data  filea  for  file  name. 

(4)  Detailed  instructions  for  intphnnentation  of  the  Data 
Tranaiator/Verifieation  Programs. 


(5)  Instructions  for  use  of  the  VP-Routine.* 

(6)  A workload  prcx^cssing  statement,  which  is  a table 
providing  a summary  of  all  the  pertinent  information 
for  implementation  of  the  benchmark. 

(7)  Instructions  and  sample  program  for  the  extraction 
of  the  VP-Routine  from  the  population  file 

(8)  Benchmark  ii.jtructions. 

(9)  Individual  program  documentation,  including  any 
known  areas  which  may  cause  implementation  prob- 
lems. 

(10)  For  variable  length  records,  or  multiply  defined 
records,  a complete  (^OBOL  record  description  is 
given,  or  a record  layout  is  provided  together  with 
its  record  type  characteristics. 

(11)  A system  flowchart  of  the  benchmark. 

(12)  Listings  of  each  program. 

LIMITATIONS 

Even  though  the  Benchmark  Preparation  System  resolves 
many  of  the  difficulties  involved  in  program  and  data 
portability,  there  are  areas  in  which  reprogramming  will  be 
required  for  complete  conversion.  The  amount  of  repro- 
gramming depends  on  the  degree  to  which  machine  de- 
pendency has  been  imposed  onto  the  pnigram.  Data  that  is 
not  explicitly  defined,  or  ftwtures  for  which  ANSI  Standard 
COBOL  does  not  have  a direct  functional  replacement 
cannot  be  detected  by  the  sanitation  process.  The  following 
are  a few  of  the  known  programming,  COBOL  characteristics, 
or  COBOL  compiler  implementation  practices  which  have  an 
impact  on  automation  of  the  conversion  prooeas. 

(1)  Funetum*  in  (Ac  nolisr  COBOL  sowros  progntm  which 
cannot  be  directly  replaced  by  featwree  or  elemente  of 
the  AS8I  language  epeeification.  Such  an  example 
would  be  the  READY  TRACE  statement  or  the 
TRANSFORM  verb  in  IBM  8ystem/3<IO  COBOL.* 
The  ANSI  language  specification  does  not  have  an 
element  or  feature  which  directly  performs  these 
functions.  To  rimulate  this  function  requires  manual 
convenaon. 
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(J)  IneompUte  or  inadequate  record  deeeriptione.  This  is 
due  to  describing  fields  or  groups  of  fields  as  alpha- 
numeric when  their  true  descriptions  could  include 
other  forms  of  data  representation.  An  example  of  this 
technique  on  the  IBM  380/370  would  be  a data  field 
describ^  as  PICTURE  X(4),  when  the  data  actually 
present  should  be  defined  as  PIC  S9(9)  COMPU- 
TATIONAL (binary),  or  a PIC  X(2)  definition  of  a 
‘ daU  field  which  is  in  fact  PIC  S9(3)  COMPUTA- 

TIONAL-3  (packed  dt^imal).  The  above  examples 
^ would  not  only  cause  the  target  compiler  to  incorrectly 

allocate  storage  but  also  would  not  provide  the  ap- 
propriate conversion  processing,  since  the  data  is 
described  as  alphanumeric. 

(3)  M uUiply  defirted  record*  which  have  different  data  etruc- 
turet  within  each  record,  and  do  not  have  a mean*  of 

I dietinquiehinq  between  record*.  The  data  conversion 

process  is  capable  of  translating  multiply  defined 
records,  but  only  if  they  can  be  identified. 

(4)  Machine  dependencies  fixed  into  the  COBOL  proqram 

I tlaeff.  This  would  include  such  things  as  assuming  the 

initial  value  of  a data  item,  initialising  numeric 
storage  areas  with  alphanumeric  literals  representing 
? a machine’s  internal  sign,  or  using  the  characwr  set 

to  represent  non-character  machine  data.  An  example 
i would  be: 

\ WORKING-STORAGE  SECTION. 

77  SIGN-FIELD  PIC  S999. 

^ 77  X-FIELD  REDEFINES  SIGN-FIELD 

PIC  XXX. 

PROCEDURE  DIVISION. 

SECTION-NAME  SECTION. 

PARAGRAPH-L. 

MOVE  -1-123  to  SIGN-FIELD 

IF  X-FIELD  EQUAL  TO  ‘12C’  GO  TO— 

Implementations  which  do  not  generate  positive  sign 
over-punches  would  require  the  procedure  to  be  modi- 
fied befoK  the  program  would  function  correctly. 

(6)  CoUtUinq  sequence  of  field*  conlaininq  alphanumeric 
data  which  are  aitical  to  proqram  proeettinq  and  whidi 
an  net  completely  defined.  This  |»oblem  is  somewhat 
reduced,  however,  in  that  COBOL  instructions  which 
may  be  affected  by  collating  sequence  are  flagged  by 
the  COBOL  to  COBOL  translator. 
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CONCLUSIONS 

The  Benchmark  Preparation  SysUm  was  developTHl  to  reduce 
the  nonportability  and  expens<‘  of  using  natural  Ismchmarks 
without  losing  the  characteristics  of  the  users  workload  in 
terms  of  processing  efficiency  and  representation.  The  results 
we  have  obtained  indicate  that  these  objectives  can  be  met. 

Our  current  test  lad  is  a Navy  benchmark  containing 
38  COBOL  programs  consisting  of  some  80,000  line's  of 
source  code,  and  includes  some  LW  data  files.  The  native 
system  is  an  IB.M  360/.50  and  the  target  machines  are  a 
UNIVAC  1108  and  HIS  OO.'iO.  These  preigrams  and  data 
files  have  beaen  successfully  conve-rted  te>  berth  the  UNIVAC.' 
1108  and  HIS  8050.  Preparation  erf  the-  benchmark  preegrams, 
development  of  data  traislaterr/verificatiern  programs,  and 
the  packaging  of  these  were  done  em  a UNIVAC  1108. 
Approximately  98  percent  of  the  changes  made  to  the 
programs  were  handle'd  by  this  system.  The  remaining  change's 
(manual)  wen:  neM;e88itated  by  extensiem  features  with  ner 
counterparts  in  the  ANSI  COBOL  standard.  Generatiern  of 
the  data  translators  and  sanitatiem  of  the  benchmark  prer- 
grams  for  packaging  required  approximately  twee  compute'r 
runs  and  one  man  hour  of  effort  per  benchmark  prergram. 
The  effort  required  on  e'ach  system  to  set  up  the  VP-Reeutine, 
and  cle!anly  compile  the  programs  has  beeen  averaging  one'- 
tenth  man  hour  per  program.  C'haracter  cede  translatieen 
posed  no  problem,  as  each  system  had  job  control  card 
options  for  transliteration  (i.e.,  EBCDIC  to  BCD  on  the 
IBM/380  and  BCD  to  FIELDATA  on  the  UNIVAC  1108 
and  IBMC  code  to  HIS  8000  code). 

Based  on  our  efforts,  we  believe  that  portability  can  be 
achieved  by  an  automated  means  without  sacrificing  the 
efficiency  of  a computer  system. 
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